Skip to content

Mixed-precision FP8 × INT4 (W4A8) inference quantizes activations (A) to float8_e4m3fn or float8_e5m2 and weights (B) to int4 while executing on the FP8 matrix unit with float32 accumulation. Previously, gemm_quantized accepted only a single quant_dtype applied identically to both operands. - #164

Merged
copybara-service[bot] merged 1 commit into
mainfrom
test_1022545780
Nov 18, 2026

Conversation

@copybara-service

@copybara-service copybara-service Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Mixed-precision FP8 × INT4 (W4A8) inference quantizes activations (A) to float8_e4m3fn or float8_e5m2 and weights (B) to int4 while executing on the FP8 matrix unit with float32 accumulation. Previously, gemm_quantized accepted only a single quant_dtype applied identically to both operands.

  • Add rhs_quant_dtype to QuantizedGemmParams in benchmarks/matmul.py (defaulting to quant_dtype, including across with_generation_defaults(), and accepting "int4" when quant_dtype is "float8_e4m3fn" or "float8_e5m2").
  • Quantize B to rhs_quant_dtype in QuantizedGemmBenchmark.setup() while keeping float32 accumulation and the quant_dtype MXU roofline ceiling whenever A is FP8.
  • Include rhs_quant_dtype in QuantizedGemmBenchmark.REPORT_SCHEMA and append _rhs_<rhs_quant_dtype> in get_run_identifier() when rhs_quant_dtype differs from quant_dtype.

@google-cla

google-cla Bot commented Nov 16, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@copybara-service
copybara-service Bot force-pushed the test_1022545780 branch 5 times, most recently from 3e24a57 to 35dd7d3 Compare November 18, 2026 03:18
… (`A`) to `float8_e4m3fn` or `float8_e5m2` and weights (`B`) to `int4` while executing on the `FP8` matrix unit with `float32` accumulation. Previously, `gemm_quantized` accepted only a single `quant_dtype` applied identically to both operands.

*   Add `rhs_quant_dtype` to `QuantizedGemmParams` in `benchmarks/matmul.py` (defaulting to `quant_dtype`, including across `with_generation_defaults()`, and accepting `"int4"` when `quant_dtype` is `"float8_e4m3fn"` or `"float8_e5m2"`).
*   Quantize `B` to `rhs_quant_dtype` in `QuantizedGemmBenchmark.setup()` while keeping `float32` accumulation and the `quant_dtype` MXU roofline ceiling whenever `A` is `FP8`.
*   Include `rhs_quant_dtype` in `QuantizedGemmBenchmark.REPORT_SCHEMA` and append `_rhs_<rhs_quant_dtype>` in `get_run_identifier()` when `rhs_quant_dtype` differs from `quant_dtype`.

PiperOrigin-RevId: 1024254668
@copybara-service
copybara-service Bot merged commit cf257b7 into main Nov 18, 2026
1 of 2 checks passed
@copybara-service
copybara-service Bot deleted the test_1022545780 branch November 18, 2026 03:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants